{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcddce7b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# download presidio\n",
    "!pip install presidio_analyzer presidio_anonymizer\n",
    "!pip install openai pandas\n",
    "!python -m spacy download en_core_web_lg"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3345f1c4",
   "metadata": {},
   "source": [
    "###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/synth_data_with_openai.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/synth_data_with_openai.ipynb)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "21410c42",
   "metadata": {},
   "source": [
    "# Use Presidio + OpenAI to turn real text into fake text\n",
    "\n",
    "This notebook uses Presidio to turn text with PII into text where PII entities are replaced with placeholders, e.g. \"`My name is David`\" turns into \"`My name is {{PERSON}}`\". Then, it calls the OpenAI API to create a fake record which is based on the original one.\n",
    "\n",
    "\n",
    "Flow:\n",
    "1. `My friend David lives in Paris. He likes it.`\n",
    "1. `My friend {{PERSON}} lives in {{CITY}}. He likes it.`\n",
    "1. `My friend Lucy lives in Beirut. She likes it.`\n",
    "    \n",
    "Note that OpenAI completion models could possibly detect PII values and replace them in one call, but it is suggested to validate that all PII entities are indeed detected.\n",
    "\n",
    "## Imports and set up OpenAI Key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "dc5146e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pprint\n",
    "from dotenv import load_dotenv\n",
    "import os\n",
    "import pandas as pd\n",
    "from openai import OpenAI\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "client = OpenAI(\n",
    "    api_key=os.environ.get(\"OPENAI_API_KEY\"),\n",
    ")\n",
    "#Or put explicitly in notebook. Find out more here: https://help.openai.com/en/articles/4936850-where-do-i-find-my-secret-api-key"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7c793237",
   "metadata": {},
   "source": [
    "## Define request for the OpenAI service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a15b64db",
   "metadata": {},
   "outputs": [],
   "source": [
    "def call_completion_model(prompt:str, model:str=\"gpt-3.5-turbo\", max_tokens:int=512) ->str:\n",
    "    \"\"\"Creates a request for the OpenAI Completion service and returns the response.\n",
    "    \n",
    "    :param prompt: The prompt for the completion model\n",
    "    :param model: OpenAI model name\n",
    "    :param max_tokens: Model's max tokens parameter\n",
    "    \"\"\"\n",
    "\n",
    "    completion = client.chat.completions.create(\n",
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": prompt,\n",
    "        }\n",
    "    ],\n",
    "    model=model,\n",
    ")\n",
    "\n",
    "    return completion.choices[0].message.content"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b0f30aed",
   "metadata": {},
   "source": [
    "## De-identify data using Presidio Analyzer and Anonymizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d0dd7820",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Hello, my name is <PERSON> and I live in <LOCATION>.\n",
      "My credit card number is <CREDIT_CARD> and my crypto wallet id is <CRYPTO>.\n",
      "\n",
      "On <DATE_TIME> I visited <URL> and sent an email to <EMAIL_ADDRESS>,  from the IP <IP_ADDRESS>.\n",
      "\n",
      "My passport: <US_PASSPORT> and my phone number: <PHONE_NUMBER>.\n",
      "\n",
      "This is a valid International Bank Account Number: <IBAN_CODE> . Can you please check the status on bank account <US_BANK_NUMBER>?\n",
      "\n",
      "<PERSON>'s social security number is <US_SSN>.  Her driver license? it is <US_DRIVER_LICENSE>.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from presidio_analyzer import AnalyzerEngine\n",
    "from presidio_anonymizer import AnonymizerEngine\n",
    "\n",
    "analyzer = AnalyzerEngine()\n",
    "anonymizer = AnonymizerEngine()\n",
    "\n",
    "sample = \"\"\"\n",
    "Hello, my name is David Johnson and I live in Maine.\n",
    "My credit card number is 4095-2609-9393-4932 and my crypto wallet id is 16Yeky6GMjeNkAiNcBY7ZhrLoMSgg1BoyZ.\n",
    "\n",
    "On September 18 I visited microsoft.com and sent an email to test@presidio.site,  from the IP 192.168.0.1.\n",
    "\n",
    "My passport: 191280342 and my phone number: (212) 555-1234.\n",
    "\n",
    "This is a valid International Bank Account Number: IL150120690000003111111 . Can you please check the status on bank account 954567876544?\n",
    "\n",
    "Kate's social security number is 078-05-1126.  Her driver license? it is 1234567A.\n",
    "\"\"\"\n",
    "\n",
    "results = analyzer.analyze(sample, language=\"en\")\n",
    "anonymized = anonymizer.anonymize(text=sample, analyzer_results=results)\n",
    "anonymized_text = anonymized.text\n",
    "print(anonymized_text)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f77b0c0f",
   "metadata": {},
   "source": [
    "## Create prompt (instructions + text to manipulate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d2a63fda",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_prompt(anonymized_text: str) -> str:\n",
    "    \"\"\"\n",
    "    Create the prompt with instructions to GPT-3.\n",
    "    \n",
    "    :param anonymized_text: Text with placeholders instead of PII values, e.g. My name is <PERSON>.\n",
    "    \"\"\"\n",
    "\n",
    "    prompt = f\"\"\"\n",
    "    Your role is to create synthetic text based on de-identified text with placeholders instead of Personally Identifiable Information (PII).\n",
    "    Replace the placeholders (e.g. ,<PERSON>, {{DATE}}, {{ip_address}}) with fake values.\n",
    "    Instructions:\n",
    "    a. Use completely random numbers, so every digit is drawn between 0 and 9.\n",
    "    b. Use realistic names that come from diverse genders, ethnicities and countries.\n",
    "    c. If there are no placeholders, return the text as is.\n",
    "    d. Keep the formatting as close to the original as possible.\n",
    "    e. If PII exists in the input, replace it with fake values in the output.\n",
    "    f. Remove whitespace before and after the generated text\n",
    "    \n",
    "    input: [[TEXT STARTS]] How do I change the limit on my credit card {{credit_card_number}}?[[TEXT ENDS]]\n",
    "    output: How do I change the limit on my credit card 2539 3519 2345 1555?\n",
    "    input: [[TEXT STARTS]]<PERSON> was the chief science officer at <ORGANIZATION>.[[TEXT ENDS]]\n",
    "    output: Katherine Buckjov was the chief science officer at NASA.\n",
    "    input: [[TEXT STARTS]]Cameroon lives in <LOCATION>.[[TEXT ENDS]]\n",
    "    output: Vladimir lives in Moscow.\n",
    "    \n",
    "    input: [[TEXT STARTS]]{anonymized_text}[[TEXT ENDS]]\n",
    "    output:\"\"\"\n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1c02d0df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the prompt with de-identified values:\n",
      "\n",
      "    Your role is to create synthetic text based on de-identified text with placeholders instead of Personally Identifiable Information (PII).\n",
      "    Replace the placeholders (e.g. ,<PERSON>, {DATE}, {ip_address}) with fake values.\n",
      "    Instructions:\n",
      "    a. Use completely random numbers, so every digit is drawn between 0 and 9.\n",
      "    b. Use realistic names that come from diverse genders, ethnicities and countries.\n",
      "    c. If there are no placeholders, return the text as is.\n",
      "    d. Keep the formatting as close to the original as possible.\n",
      "    e. If PII exists in the input, replace it with fake values in the output.\n",
      "    f. Remove whitespace before and after the generated text\n",
      "    \n",
      "    input: [[TEXT STARTS]] How do I change the limit on my credit card {credit_card_number}?[[TEXT ENDS]]\n",
      "    output: How do I change the limit on my credit card 2539 3519 2345 1555?\n",
      "    input: [[TEXT STARTS]]<PERSON> was the chief science officer at <ORGANIZATION>.[[TEXT ENDS]]\n",
      "    output: Katherine Buckjov was the chief science officer at NASA.\n",
      "    input: [[TEXT STARTS]]Cameroon lives in <LOCATION>.[[TEXT ENDS]]\n",
      "    output: Vladimir lives in Moscow.\n",
      "    \n",
      "    input: [[TEXT STARTS]]\n",
      "Hello, my name is <PERSON> and I live in <LOCATION>.\n",
      "My credit card number is <CREDIT_CARD> and my crypto wallet id is <CRYPTO>.\n",
      "\n",
      "On <DATE_TIME> I visited <URL> and sent an email to <EMAIL_ADDRESS>,  from the IP <IP_ADDRESS>.\n",
      "\n",
      "My passport: <US_PASSPORT> and my phone number: <PHONE_NUMBER>.\n",
      "\n",
      "This is a valid International Bank Account Number: <IBAN_CODE> . Can you please check the status on bank account <US_BANK_NUMBER>?\n",
      "\n",
      "<PERSON>'s social security number is <US_SSN>.  Her driver license? it is <US_DRIVER_LICENSE>.\n",
      "[[TEXT ENDS]]\n",
      "    output:\n"
     ]
    }
   ],
   "source": [
    "print(\"This is the prompt with de-identified values:\")\n",
    "print(create_prompt(anonymized_text))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "49aabd81",
   "metadata": {},
   "source": [
    "## Call the LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "23653b0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt_res = call_completion_model(create_prompt(anonymized_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "561f0565",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello, my name is Aaliyah and I live in Tokyo.\n",
      "My credit card number is 4928 7562 1034 8907 and my crypto wallet id is 0x3B 7a 5f 1C.\n",
      "\n",
      "On 02/07/2023 15:45 I visited www.example.com and sent an email to example@email.com,  from the IP 127.0.0.1.\n",
      "\n",
      "My passport: L921483B and my phone number: +1 (555) 123-4567.\n",
      "\n",
      "This is a valid International Bank Account Number: FR76 1234 5789 1256 3321 7564 901. Can you please check the status on bank account 987654321?\n",
      "\n",
      "Eliana's social security number is 123-45-6789.  Her driver license? it is DL12345678.\n"
     ]
    }
   ],
   "source": [
    "print(gpt_res)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cdf4dbe1",
   "metadata": {},
   "source": [
    "### Alternatively, run on a list of template sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "234b6204",
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib\n",
    "\n",
    "templates = []\n",
    "\n",
    "url = \"https://raw.githubusercontent.com/microsoft/presidio-research/master/presidio_evaluator/data_generator/raw_data/templates.txt\"\n",
    "for line in urllib.request.urlopen(url):\n",
    "    templates.append(line.decode('utf-8')) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "da1773fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example templates:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['I want to increase limit on my card # {{credit_card_number}} for certain duration of time. is it possible?\\n',\n",
       " 'My credit card {{credit_card_number}} has been lost, Can I request you to block it.\\n',\n",
       " 'Need to change billing date of my card {{credit_card_number}}\\n',\n",
       " 'I want to update my primary and secondary address to the same: {{address}}\\n',\n",
       " \"In case of my child's account, we need to add {{person}} as guardian\\n\"]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Example templates:\")\n",
    "templates[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3757461b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{ 'original': 'I want to increase limit on my card # {{credit_card_number}} for certain duration of time. is '\n",
      "              'it possible?\\n',\n",
      "  'synthetic': 'I want to increase limit on my card # 4701 2895 7462 8306 for certain duration of time. is '\n",
      "               'it possible?'}\n",
      "--------------\n",
      "{ 'original': 'My credit card {{credit_card_number}} has been lost, Can I request you to block it.\\n',\n",
      "  'synthetic': 'My credit card 4892 7634 1023 8756 has been lost, Can I request you to block it.'}\n",
      "--------------\n",
      "{ 'original': 'Need to change billing date of my card {{credit_card_number}}\\n',\n",
      "  'synthetic': 'Need to change billing date of my card 4876 2035 6981 7423'}\n",
      "--------------\n",
      "{ 'original': 'I want to update my primary and secondary address to the same: {{address}}\\n',\n",
      "  'synthetic': 'I want to update my primary and secondary address to the same: 123 Main Street, Apt 4.'}\n",
      "--------------\n",
      "{ 'original': \"In case of my child's account, we need to add {{person}} as guardian\\n\",\n",
      "  'synthetic': \"In case of my child's account, we need to add Abdul as guardian\"}\n",
      "--------------\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------\n",
      "{ 'original': '{{name}} lives at {{building_number}} {{street_name}}, {{city}}\\n',\n",
      "  'synthetic': 'John Smith lives at 635 Poplar Street, Houston'}\n",
      "--------------\n",
      "{ 'original': '{{first_name_male}} had given {{first_name}} his address: {{building_number}} '\n",
      "              '{{street_name}}\\n',\n",
      "  'synthetic': 'Adam had given Sarah his address: 44 Apple Street'}\n",
      "--------------\n",
      "{ 'original': '{{first_name_male}} had given {{first_name}} his address: {{building_number}} '\n",
      "              '{{street_name}}, {{city}}\\n',\n",
      "  'synthetic': 'David had given Emma his address: 515 Elm Street, Camden.'}\n",
      "--------------\n",
      "{ 'original': 'What is your address? it is {{address}}\\n',\n",
      "  'synthetic': 'What is your address? it is 3498 Allensby Street, Los Angeles, CA 90011.'}\n",
      "--------------\n",
      "{'original': 'We moved here from {{city}}\\n', 'synthetic': 'We moved here from Paris.'}\n",
      "--------------\n",
      "{'original': 'We moved here from {{country}}\\n', 'synthetic': 'We moved here from Venezuela.'}\n",
      "--------------\n",
      "{ 'original': '{{person}}\\\\n\\\\n{{building_number}} {{street_name}}\\\\n {{secondary_address}}\\\\n {{city}}\\\\n '\n",
      "              '{{country}} {{postcode}}\\\\n{{phone_number}}-Office\\\\,{{phone_number}}-Fax\\n',\n",
      "  'synthetic': 'Jessica Thompson\\n'\n",
      "               '    8745 West Drive\\n'\n",
      "               '    Suite 1402\\n'\n",
      "               '    Brooklyn, NY USA 12009\\n'\n",
      "               '    789-534-9921-Office, 567-945-0023-Fax'}\n",
      "--------------\n",
      "{ 'original': '{{person}}\\\\n{{job}}\\\\n{{organization}}\\\\n{{address}}\\n',\n",
      "  'synthetic': 'John Smith\\\\nAccountant\\\\nGlobalTech Solutions\\\\n25 Speedwell Street, Richmond, VA 23223'}\n",
      "--------------\n",
      "{ 'original': 'Our offices are located at {{address}}\\n',\n",
      "  'synthetic': 'Our offices are located at 1234 Main St, Los Angeles, CA 91234.'}\n",
      "--------------\n",
      "{ 'original': 'Please return to {{address}} in case of an issue.\\n',\n",
      "  'synthetic': 'Please return to 123 Cherry Street, El Monte, CA 91731 in case of an issue.'}\n",
      "--------------\n",
      "{ 'original': '{{organization}}\\\\n\\\\n{{address}}\\n',\n",
      "  'synthetic': 'ABC Inc.\\n     1234 Main Street, Anytown, ST 12345'}\n",
      "--------------\n",
      "{ 'original': 'The {{organization}} office is at {{address}}\\n',\n",
      "  'synthetic': 'The ABC Corporation office is at 123 Redwood Street, Anytown, USA.'}\n",
      "--------------\n",
      "{ 'original': '{{name}}\\\\n{{organization}}\\\\n{{address}}\\\\n{{phone_number}} office\\\\n{{phone_number}} '\n",
      "              'fax\\\\n{{phone_number}} mobile\\\\n\\n',\n",
      "  'synthetic': 'Larry Fernandez\\\\nStar Enterprises\\\\n421 5th Avenue, Los Angeles, CA 90012\\\\n123-456-7890 '\n",
      "               'office\\\\n456-789-0123 fax\\\\n256-454-2397 mobile\\\\n'}\n",
      "--------------\n",
      "{ 'original': '{{name}}\\\\n{{organization}}\\\\n{{address}}\\\\nMobile: {{phone_number}}\\\\nDesk: '\n",
      "              '{{phone_number}}\\\\nFax: {{phone_number}}\\\\n\\n',\n",
      "  'synthetic': 'John Brown\\n'\n",
      "               '    ABC Consulting\\n'\n",
      "               '    5th St., Suite 116, LA CA 90004\\n'\n",
      "               '    Mobile: 213-294-4497\\n'\n",
      "               '    Desk: 424-348-1275\\n'\n",
      "               '    Fax: 323-456-2545'}\n",
      "--------------\n",
      "{ 'original': 'Billing address: {{name}}\\\\n    {{building_number}} {{street_name}} '\n",
      "              '{{secondary_address}}\\\\n   {{city}}\\\\n    {{state_abbr}}\\\\n    {{zipcode}}\\\\n\\n',\n",
      "  'synthetic': 'Billing address: Mariam Rajput\\n'\n",
      "               '    576 Broadway Street Apartment A8\\n'\n",
      "               '   Sacramento\\n'\n",
      "               '    CA\\n'\n",
      "               '    95349'}\n",
      "--------------\n",
      "{ 'original': \"As promised, here's {{first_name}}'s address:\\\\n\\\\n{{address}}\\n\",\n",
      "  'synthetic': \"As promised, here's Aamir's address:\\\\n\\\\n2166 Sesame Street, Fortaleza, Ceará, Brazil.\"}\n",
      "--------------\n",
      "{ 'original': '>{{name}}\\\\n>{{organization}}\\\\n>{{person}}\\\\n>{{building_number}} '\n",
      "              '{{street_name}}\\\\n>{{secondary_address}}\\\\n>{{city}}\\\\n>{{country}} {{postcode}}\\n',\n",
      "  'synthetic': '>Freda Chen\\n'\n",
      "               '>Example Inc.\\n'\n",
      "               '>John Doe\\n'\n",
      "               '>50 Main Street\\n'\n",
      "               '>Apt. 2\\n'\n",
      "               '>New York\\n'\n",
      "               '>United States 10005'}\n",
      "--------------\n",
      "{ 'original': '??? {{name}}\\\\n??? {{organization}}\\\\n??? {{building_number}} {{street_name}}\\\\n??? '\n",
      "              '{{secondary_address}}\\\\n??? {{city}}\\\\n??? {{country}} {{postcode}}\\n',\n",
      "  'synthetic': 'John Smith\\n'\n",
      "               '     ABC Corporation\\n'\n",
      "               '     192 Main Street\\n'\n",
      "               '     Suite 100\\n'\n",
      "               '     Austin\\n'\n",
      "               '     United States 78701'}\n",
      "--------------\n",
      "{ 'original': '> \\\\n> {{name}}\\\\n> {{organization}}\\\\n> {{person}}\\\\n> {{building_number}} '\n",
      "              '{{street_name}}\\\\n> {{secondary_address}}\\\\n> {{city}}\\\\n> {{country}} {{postcode}}\\n',\n",
      "  'synthetic': '> John Doe\\n'\n",
      "               '    > ABC Corp\\n'\n",
      "               '    > Jane Smith\\n'\n",
      "               '    > 123 Main Street\\n'\n",
      "               '    > APT 8\\n'\n",
      "               '    > San Francisco\\n'\n",
      "               '    > United States 12345'}\n",
      "--------------\n",
      "{ 'original': 'Pedestrians must enter on {{street_name}} St. the first three months\\n',\n",
      "  'synthetic': 'Pedestrians must enter on Jericho Avenue St. the first three months'}\n",
      "--------------\n",
      "{ 'original': 'When: {{date_time}}\\\\nWhere: {{city}} Country Club.\\n',\n",
      "  'synthetic': 'When: 05/01/2020 10:00am\\\\nWhere: Richmond Country Club.'}\n",
      "--------------\n",
      "{ 'original': \"We'll meet {{day_of_week}} at {{organization}}, {{building_number}} {{street_name}}, \"\n",
      "              '{{city}}\\n',\n",
      "  'synthetic': \"We'll meet Monday at Smartdel Solutions, 145 King Street, San Diego.\"}\n",
      "--------------\n",
      "{ 'original': 'They had 6: {{first_name}}, {{first_name}}, {{first_name}}, {{first_name}}, {{first_name}} '\n",
      "              'and {{first_name}}.\\n',\n",
      "  'synthetic': 'They had 6: Sarah, Micheal, Kanak, Hana, Mei and Dan.'}\n",
      "--------------\n",
      "{'original': 'She moved here from {{country}}\\n', 'synthetic': 'She moved here from Mexico.'}\n",
      "--------------\n",
      "{'original': 'My zip code is {{zipcode}}\\n', 'synthetic': 'My zip code is 47713.'}\n",
      "--------------\n",
      "{'original': 'ZIP: {{zipcode}}\\n', 'synthetic': 'ZIP: 08547'}\n",
      "--------------\n",
      "{'original': 'The bus station is on {{street_name}}\\n', 'synthetic': 'The bus station is on Wilson Avenue.'}\n",
      "--------------\n",
      "{ 'original': \"They're not answering at {{phone_number}}\\n\",\n",
      "  'synthetic': \"They're not answering at 654-339-1013.\"}\n",
      "--------------\n",
      "{ 'original': 'God gave rock and roll to you, gave rock and roll to you, put it in the soul of everyone.\\n',\n",
      "  'synthetic': 'God gave rock and roll to you, gave rock and roll to you, put it in the soul of everyone.'}\n",
      "--------------\n",
      "{'original': '3... 2... 1... liftoff!\\n', 'synthetic': '3... 2... 1... liftoff!'}\n",
      "--------------\n",
      "{ 'original': 'My great great grandfather was called {{name_male}}, and my great great grandmother was '\n",
      "              'called {{name_female}}\\n',\n",
      "  'synthetic': 'My great great grandfather was called Michael, and my great great grandmother was called '\n",
      "               'Emma.'}\n",
      "--------------\n",
      "{'original': 'She named him {{first_name_male}}\\n', 'synthetic': 'She named him Juan.'}\n",
      "--------------\n",
      "{ 'original': 'Name:    {{name}}\\\\nAddress:     {{address}}\\n',\n",
      "  'synthetic': 'Name:    Amari Walters\\\\nAddress:     32 Webster Street, Salem, MA 01819'}\n",
      "--------------\n",
      "{ 'original': 'Follow up with {{name}} in a couple of months.\\n',\n",
      "  'synthetic': 'Follow up with Beatriz Lawrence in a couple of months.'}\n",
      "--------------\n",
      "{ 'original': '{{prefix_male}} {{last_name_male}} is a {{age}} year old man who grew up in {{city}}.\\n',\n",
      "  'synthetic': 'Mr.Williams is a 28 year old man who grew up in Dallas.'}\n",
      "--------------\n",
      "{ 'original': 'Date: {{date_time}}\\\\nName: {{name}}\\\\nPhone: {{phone_number}}\\n',\n",
      "  'synthetic': 'Date: 01/03/2021 13:45 \\\\nName: Pratima Joshi \\\\nPhone: 467-562-8954'}\n",
      "--------------\n",
      "{ 'original': '{{first_name}}: \"Who are you?\"\\\\n{{first_name_female}}:\"I\\'m {{first_name}}\\'s daughter\".\\n',\n",
      "  'synthetic': 'Bob: \"Who are you?\"\\\\nMaria:\"I\\'m Bob\\'s daughter\".'}\n",
      "--------------\n",
      "{ 'original': 'At my suggestion, one morning over breakfast, she agreed, and on the last Sunday before Labor '\n",
      "              'Day we returned to {{city}} by helicopter.\\n',\n",
      "  'synthetic': 'At my suggestion, one morning over breakfast, she agreed, and on the last Sunday before '\n",
      "               'Labor Day we returned to Paris by helicopter.'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------\n",
      "{ 'original': \"It was a done thing between him and {{first_name}}'s kid; and everybody thought so.\\n\",\n",
      "  'synthetic': \"It was a done thing between him and Jeffery's kid; and everybody thought so.\"}\n",
      "--------------\n",
      "{ 'original': 'Capitalized words like Wisdom and Discipline are often mistaken with names.\\n',\n",
      "  'synthetic': 'Capitalized words like Wisdom and Discipline are often mistaken with names.'}\n",
      "--------------\n",
      "{ 'original': 'The letter arrived at {{address}} last night.\\n',\n",
      "  'synthetic': 'The letter arrived at 1143 Orange Street last night.'}\n",
      "--------------\n",
      "{ 'original': 'The Princess Royal arrived at {{city}} this morning from {{country}}.\\n',\n",
      "  'synthetic': 'The Princess Royal arrived at London this morning from France.'}\n",
      "--------------\n",
      "{'original': \"I'm in {{city}}, at the conference\\n\", 'synthetic': \"I'm in Toronto, at the conference.\"}\n",
      "--------------\n",
      "{ 'original': '{{name}}, the {{job}}, said: \"I\\'m glad to hear that this has been withdrawn – quite why they '\n",
      "              'thought this would go down well is beyond me.\"\\n',\n",
      "  'synthetic': 'Gloria Green, the Nurse Practitioner, said: \"I\\'m glad to hear that this has been withdrawn '\n",
      "               '– quite why they thought this would go down well is beyond me.\"'}\n",
      "--------------\n",
      "{ 'original': '\"I\\'m glad to hear that {{country}} is moving in that direction,\" says {{last_name}}.\\n',\n",
      "  'synthetic': '\"I\\'m glad to hear that Canada is moving in that direction,\" says Smith.'}\n",
      "--------------\n",
      "{ 'original': 'I am {{nation_woman}} but I live in {{country}}.\\n',\n",
      "  'synthetic': 'I am Marianna Montenegro but I live in Ukraine.'}\n",
      "--------------\n",
      "{'original': 'We are proud {{nation_plural}}\\n', 'synthetic': 'We are proud Americans.'}\n",
      "--------------\n",
      "{ 'original': \"{{person}}'s killers sentenced to life in prison\\n\",\n",
      "  'synthetic': \"John Smith's killers sentenced to life in prison\"}\n",
      "--------------\n",
      "{ 'original': \"{{country}} leader gives 'kill without warning' order\\n\",\n",
      "  'synthetic': \"Brazilian leader gives 'kill without warning' order\"}\n",
      "--------------\n",
      "{ 'original': 'The {{nationality}} Border Force have detained top-flight tennis player {{name_female}} over '\n",
      "              'visa disputes.\\n',\n",
      "  'synthetic': 'The British Border Force have detained top-flight tennis player Maria Rodriguez over visa '\n",
      "               'disputes.'}\n",
      "--------------\n",
      "{ 'original': 'You will be responsible for the husbandry and care of a large variety of species including '\n",
      "              'lemurs, antelope, camels, and more\\n',\n",
      "  'synthetic': 'You will be responsible for the husbandry and care of a large variety of species including '\n",
      "               'lemurs, antelope, camels, and more.'}\n",
      "--------------\n",
      "{ 'original': '{{name}}\\\\n\\\\n{{job}}\\\\n\\\\nPersonal '\n",
      "              'Info:\\\\nPhone:\\\\n{{phone_number}}\\\\n\\\\nE-mail:\\\\n{{email}}\\\\n\\\\nWebsite:\\\\n{{url}}\\\\n\\\\nAddress:\\\\n{{address}}.\\n',\n",
      "  'synthetic': 'Robert James\\\\n\\\\nSoftware Engineer\\\\n\\\\nPersonal '\n",
      "               'Info:\\\\nPhone:\\\\n555-847-8915\\\\n\\\\nE-mail:\\\\nrobertjames@example.com\\\\n\\\\nWebsite:\\\\nwww.example.com\\\\n\\\\nAddress:\\\\n277 '\n",
      "               'Park Ave North, Denver, CO 80100.'}\n",
      "--------------\n",
      "{ 'original': '{{name}}\\\\n\\\\n{{city}}\\\\n{{country}}\\n',\n",
      "  'synthetic': 'John Smith\\n     Los Angeles\\n     United States'}\n",
      "--------------\n",
      "{ 'original': 'Title VII of the Civil Rights Act of {{year}} protects individuals against employment '\n",
      "              'discrimination on the basis of race and color as well as national origin, sex, or religion.\\n',\n",
      "  'synthetic': 'Title VII of the Civil Rights Act of 1964 protects individuals against employment '\n",
      "               'discrimination on the basis of race and color as well as national origin, sex, or religion.'}\n",
      "--------------\n",
      "{ 'original': 'Energetic and driven salesperson with 8+ years of professional experience in inbound and '\n",
      "              'outbound sales. Awarded Salesperson of the Month three times. Helped increase inbound sales '\n",
      "              'by 16% within the first year of employment. Looking to support {{organization}} in {{city}} '\n",
      "              '{{zipcode}} in its mission to become a market-leading solution.\\n',\n",
      "  'synthetic': 'Energetic and driven salesperson with 8+ years of professional experience in inbound and '\n",
      "               'outbound sales. Awarded Salesperson of the Month three times. Helped increase inbound sales '\n",
      "               'by 16% within the first year of employment. Looking to support Acme Corporation in Los '\n",
      "               'Angeles 90018 in its mission to become a market-leading solution.'}\n",
      "--------------\n",
      "{ 'original': 'The bus drops you off at {{building_number}} {{street_name}} St.\\n',\n",
      "  'synthetic': 'The bus drops you off at 2774 Chestnut St.'}\n",
      "--------------\n",
      "{ 'original': 'Ask the driver to stop at the corner of {{street_name}} St. and {{street_name}} St.\\n',\n",
      "  'synthetic': 'Ask the driver to stop at the corner of Maple St. and Sycamore St.'}\n",
      "--------------\n",
      "{ 'original': 'He lives on the north side of {{street_name}}.\\n',\n",
      "  'synthetic': 'He lives on the north side of Abbey Road.'}\n",
      "--------------\n",
      "{ 'original': \"I used to work for {{organization}} as {{job}}, but quit a few months ago. Now I'm \"\n",
      "              'unemployed.\\n',\n",
      "  'synthetic': \"I used to work for ABC Corporation as Software Engineer, but quit a few months ago. Now I'm \"\n",
      "               'unemployed.'}\n",
      "--------------\n",
      "{'original': '{{city}} bridge is falling down.\\n', 'synthetic': 'Berlin bridge is falling down.'}\n",
      "--------------\n",
      "{ 'original': '{{name}} of {{organization}} is the CEO of the year. ABC Business considered several other '\n",
      "              \"influential CEOs for this year's honor, including {{name}} of {{organization}}, \"\n",
      "              \"{{organization}}'s {{name}}, {{name}} of {{organization}}'s, {{name}} of {{organization}}, \"\n",
      "              \"and {{organization}}'s {{name}}.\\n\",\n",
      "  'synthetic': 'Franklin Smith of Technology Solutions International is the CEO of the year. ABC Business '\n",
      "               \"considered several other influential CEOs for this year's honor, including Madison Chang of \"\n",
      "               \"Radiance Digital, Radiance Digital's Ashleigh Jones, Cash Huang of Clark & Partner's, Sierra \"\n",
      "               \"Urbina of Intelicity, and Clark & Partners's Asher Kenney.\"}\n",
      "--------------\n",
      "{ 'original': '{{organization}} is a design agency based in {{city}}.\\n',\n",
      "  'synthetic': 'Maestro Design Inc. is a design agency based in Amsterdam.'}\n",
      "--------------\n",
      "{ 'original': 'Action & Adventure, Animation, Comedy, Kids & Family, Mystery & Suspense\\\\nDirected By:    '\n",
      "              '{{name}}\\n',\n",
      "  'synthetic': 'Action & Adventure, Animation, Comedy, Kids & Family, Mystery & Suspense\\n'\n",
      "               'Directed By: Esther Jones'}\n",
      "--------------\n",
      "{ 'original': '{{first_name}}: What a wife.\\\\n{{first_name}}: Remember me, {{first_name}}? When I killed '\n",
      "              'your brother, I talked just like this!\\\\n{{first_name}}: You saved my life! How can I ever '\n",
      "              'repay you?\\n',\n",
      "  'synthetic': 'Emma: What a wife.\\n'\n",
      "               '    Emma: Remember me, Emma? When I killed your brother, I talked just like this!\\n'\n",
      "               '    Emma: You saved my life! How can I ever repay you?'}\n",
      "--------------\n",
      "{'original': 'He just turned {{age}} years old\\n', 'synthetic': 'He just turned 7 years old'}\n",
      "--------------\n",
      "{ 'original': \"I'm {{name}}, originally from {{city}}, and i'm {{age}} y/o.\\n\",\n",
      "  'synthetic': \"I'm Emily Evanston, originally from London, and I'm 24 y/o.\"}\n",
      "--------------\n",
      "{ 'original': 'Patient is a {{age}}-year-old male with a history of headaches\\n',\n",
      "  'synthetic': 'Patient is a 35-year-old male with a history of headaches'}\n",
      "--------------\n",
      "{'original': 'I just turned {{age}}\\n', 'synthetic': 'I just turned 24.'}\n",
      "--------------\n",
      "{'original': 'My father retired at the age of {{age}}\\n', 'synthetic': 'My father retired at the age of 60.'}\n",
      "--------------\n",
      "{ 'original': 'This {{age}} year old female complaining of stomach pain.\\n',\n",
      "  'synthetic': 'This 28 year old female complaining of stomach pain.'}\n",
      "--------------\n",
      "{ 'original': \"My birthday is on the weekend. I'll turn {{age}}.\\n\",\n",
      "  'synthetic': \"My birthday is on the weekend. I'll turn 20.\"}\n",
      "--------------\n",
      "{'original': 'My brother just turned {{age}}\\n', 'synthetic': 'My brother just turned 18.'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------\n",
      "{ 'original': '{{prefix}} {{last_name}} flew to {{city}} on {{day_of_week}} morning.',\n",
      "  'synthetic': 'Dr. Nguyen flew to Los Angeles on Tuesday morning.'}\n",
      "--------------\n"
     ]
    }
   ],
   "source": [
    "templates_to_use = templates[:5]\n",
    "\n",
    "\n",
    "import time\n",
    "pp = pprint.PrettyPrinter(indent=2, width=110)\n",
    "sentences = []\n",
    "for template in templates_to_use:\n",
    "    synth_sentence = call_completion_model(create_prompt(template))\n",
    "    sentence_dict = {\"original\": template, \"synthetic\":synth_sentence.strip()}\n",
    "    sentences.append(sentence_dict)\n",
    "    pp.pprint(sentence_dict)\n",
    "    time.sleep(3) # wait to not get blocked by service (only applicable for the free tier)\n",
    "    print(\"--------------\")\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3981c913",
   "metadata": {},
   "source": [
    "This notebook demonstrates how to leverage OpenAI models for fake/surrogate data generation. It uses Presidio to first de-identify data (as de-identification might be required prior to passing the model to OpenAI), and then uses OpenAI completion models to create synthetic/fake/surrogate data based on real data. OpenAI models would also potentially remove additional PII entities, if those are not detected by Presidio.\n",
    "\n",
    "Some impressions:\n",
    "1. LLMs sometimes gives additonal output, especially if the text is a question or concerning a human/bot interaction. Engineering the prompt can mitigate some of these issues. Potential post-processing might be required.\n",
    "2. LLMs sometimes creates fake values even in the absence of placeholders.\n",
    "3. LLMs re-uses context from other sentences, which could cause phone numbers are sometimes generated using a credit card pattern or other similar mistakes.\n",
    "4. Co-references are sometimes missed (i.e. two name placeholders that should be filled with the same name, or referencing he/she to a male/female name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ff32f21-33c4-4df1-9626-73cbdf5145df",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "presidio_e2e",
   "language": "python",
   "name": "presidio"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
